Skip to content

Read report cliffs from simulation runs' cliff results, and take include_cliffs on re-runs - #4329

Draft
anth-volk wants to merge 21 commits into
v2plan/c1-37-cliff-resultsfrom
v2plan/c1-40-report-cliffs
Draft

anth-volk wants to merge 21 commits into
v2plan/c1-37-cliff-resultsfrom
v2plan/c1-40-report-cliffs

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4328

Stacks on #4327 (v2plan/c1-37-cliff-results, revision bc334f9877b7, table cliff_impacts); retarget to master once #4327 merges.

Summary

  • Cliffs come from the simulation runs. An economy report run's result.cliff_impact is its two simulation runs' succeeded cliff_impacts rows, {baseline, reform}, once both have one, and null otherwise. Nothing in API v1 reads report_runs' four cliff columns any more, so a later revision can drop them. A report run that stored cliffs only in those columns now reads none.

  • Each economy report read gives cliffs: how far the two cliff calculations have got, for the run read in any status. It appears on the get, and on the create and re-run answers, but not on lists or searches, and is null for a report with no run.

    "cliffs": {
      "status": "running",
      "baseline": {"status": "succeeded", "error_message": null},
      "reform":   {"status": "running",   "error_message": null}
    }
    • A side's status: not_calculated, pending, running, succeeded, or failed. A side is failed when its newest calculation failed and none is under way; error_message then says why.
    • The overall status: succeeded once both sides are. Otherwise it is failed if either side is, else not_calculated if either side is, else running if either side is, else pending.
    • Which row a side reads: its simulation run's calculation that hasn't failed (at most one, by uq_cliff_impacts_unfailed), else its newest failed one. One query per distinct simulation run, so a report naming one run on both sides reads it once.
  • include_cliffs on re-runs. POST /v2/reports/{report_id}/runs takes include_cliffs (default false).

    • API v1 sends "include_cliffs": true to the simulation service only when asked, so a re-run that doesn't ask still works on a simulation service whose re-run body takes only force.
    • A household report or budget window asking for cliffs answers 422 before the simulation service is asked or anything is recorded; their creates already refuse the field.
    • The economy create already sends include_cliffs and is unchanged.
  • The four cliff columns leave the ReportRun model. They are declared only in the table (__table_args__) and listed in exclude_properties, so ReportRun has no field or attribute for them and its SQL never names them. The table, ck_report_runs_cliff_impact_complete and the grants are unchanged, so there is no revision. The simulation service can keep writing the columns until its own change deploys.

  • File split. stored_reports.services reached its 800-line limit, so the code that reads reports as the requesting user sees them moves unchanged to stored_reports.requester_views.services. _requesters_view is renamed requesters_view there, because another module now imports it.

  • Changelog fragment: changelog.d/report-cliffs-from-simulation-runs.changed.md.

Merge and deploy order

  1. Store one cliff result per simulation run in the v2 database, with grants #4327's revision bc334f9877b7, applied with its grants in staging and production. This PR reads cliff_impacts as the runtime role.
  2. The simulation service's conversion of its database code to mapped classes and Sessions (C1 · 33a to C1 · 33e).
  3. The simulation service's C1 · 38, redone on that conversion, as the default worker's v2 app. It calculates and writes each run's cliff result, and stops writing the four columns. Calculate each simulation run's cliff gap and share as its own cliff result policyengine-sim-api#1096 won't merge as it stands.
  4. The simulation service's C1 · 39, redone on that conversion. It starts the cliff calculations on creates and re-runs with include_cliffs. Start each side's cliff calculation when a report asks for cliffs policyengine-sim-api#1099 won't merge as it stands.

This PR needs only step 1 to deploy safely, and doesn't depend on when steps 2 to 4 happen. Until step 4 deploys:

  • Asking for cliffs is refused. include_cliffs: true answers 422 and the whole request is refused: a create creates no report, and a re-run starts no run. On a create the simulation service answers cliffs_not_available; on a re-run, a validation error naming include_cliffs. API v1 passes either on as 422. The OpenAPI descriptions say so.
  • Reads show no cliffs. Nothing starts calculations, so every side reads not_calculated and result.cliff_impact is null.
  • Dropping the columns needs more than this PR. This release no longer depends on the four cliff columns, but it still maps reports.label, description and user_id and both trigger columns. A separate change (C1 · 40a) stops mapping those five, and must deploy before the later revision (C1 · 41) drops all nine.
  • Older stored cliffs aren't shown. A report run that stored cliffs in the four report_runs columns reads none, and they can't be calculated again until step 4. API v1's callers haven't been asking for cliffs, so few or none are expected, but production hasn't been counted.

Migration notes

  • Route group: v2 reports (/v2/reports: read, create, re-run). Nothing on the Flask path changes.
  • API contract changes, intended:
    • economy report items gain cliffs;
    • the re-run body gains include_cliffs;
    • result.cliff_impact now comes from the simulation runs' cliff results.
  • Generated documents: scripts/export_migration_contracts.py regenerates both migration contract documents unchanged; they don't record this route's answer shape.
  • Deployment: no deploy step, workflow, image, environment variable or secret changes.

Inline suppressions added

  • # noqa: PLR0913 on read_economy_result_rows, the database read of a report run's result rows. It now takes the cliff calculations the report service has already read, so a run's cliffs are read once whatever its status. Its other inputs (the session, the country, the run, its reform simulation run and the local-area flag) are each needed, and a parameter object used only by this one call would add a type without simplifying anything.
  • # noqa: PLR0913 on the integration test helper store_cliff_calculation, which takes the connection, the simulation run, the status and each column a test sets.

Testing

Each test was written first and seen failing for the expected reason, and committed with "(failing)" in the subject before the code that passes it. I ran only the tests this PR adds or changes, by node ID: 41 node IDs, 84 cases with their parameters, all passing. The PostgreSQL integration tests ran against a disposable PostgreSQL 17 database built from this branch, with reconcile_v2_privileges.py apply --create-missing-roles and verify, as the runtime role.

  • New:
    • tests/unit/v2/test_report_cliff_reads.py: each side's row; the side and overall statuses; cliff_impact; one run on both sides;
    • tests/integration/test_v2_report_cliff_reads.py: succeeded, still calculating and failed sides; the four columns not read; no run;
    • route, OpenAPI, client-body and re-run tests in test_report_read_routes.py, test_report_run_routes.py, test_simulation_report_reruns.py, test_v2_report_runs.py and test_v2_report_creation.py;
    • client tests pinning the simulation service's two current refusal bodies, in test_simulation_reports.py and test_simulation_report_reruns.py. These passed when written, because API v1 already answers both with 422.
  • Changed:
    • tests/fixtures/economy_results.py stores a result's cliffs as each simulation run's succeeded cliff_impacts row instead of the report run's columns;
    • the golden rebuild test expects no cliff impact from the report-run writer's rows.

Also run: ruff, ruff format, make typecheck-v2 (mypy), lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py and scripts/export_migration_contracts.py. alembic -c alembic-v2.ini check finds no schema change, and reconcile_v2_privileges.py apply --create-missing-roles and verify pass; there is no revision or grant change. Explicit mypy on the changed policyengine_api/data/v2/models files reports only the Field(sa_type=...) overload errors those modules already have.

Reviewed in six adversarial rounds, three for each stage of the work after the first build:

  • Round 1 (the build): no bugs. Its code-quality findings are fixed: the economy read has its own method, descriptions are corrected, and a golden test that couldn't fail is replaced by one that fails if the report run's result columns hold a cliff column again.
  • Rounds 2 and 3 (wording for a later simulation service deploy): no behaviour bugs. They found test descriptions that misstated what the simulation service's report run writer stores, and imprecise OpenAPI text. All are fixed.
  • Rounds 4 to 6 (taking the cliff columns out of ReportRun): round 4 found a description implying this release survives the whole later column drop; round 5 found a category name that would have failed the v2 model-layer rule test. Both are fixed, along with their code-quality findings, and round 6 found no bugs.

🤖 Generated with Claude Code

anth-volk and others added 7 commits November 17, 2026 19:22
…failing)

A report run's cliff impact comes from its two simulation runs' cliff_impacts
rows, not from report_runs' four cliff columns, and an economy report read
says how far each side's cliff calculation has got (cliffs). The fixtures
store a result's cliffs as each simulation run's succeeded calculation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An economy report run's cliff impact is its two simulation runs' succeeded
cliff_impacts rows, and no read uses report_runs' four cliff columns any
more, so C1 · 41 can drop them. Each side reads its run's calculation that
hasn't failed, else its newest failed one. Economy report reads (get, and
the create and re-run answers) gain cliffs: an overall status and each
side's status and error message, for a run in any status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /v2/reports/{id}/runs takes include_cliffs and passes it to the
simulation service, sending it only when asked; a household report's or
budget window's re-run asking for cliffs is refused with 422 before the
service is asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /v2/reports/{id}/runs takes include_cliffs and sends it to the
simulation service only when true, so a re-run that doesn't ask still runs
on a simulation service that doesn't take the field yet. A household
report's or budget window's re-run asking for cliffs answers 422 before the
service is asked or anything recorded. The create body's include_cliffs now
says what it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stored_reports.services reached its 800-line limit. The code that reads
reports as the requesting user sees them (their saved report's label and
their newest requested run) moves unchanged to
stored_reports.requester_views.services; _requesters_view is renamed
requesters_view there, since the report service now imports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1's quality findings: the economy read moves into
_economy_report, as household reports and budget windows have theirs; the
result's cliff_impact, the read and create routes, and a re-run of a run on
older versions now say how cliffs read and start, and that a simulation
service that doesn't calculate cliffs per simulation run yet refuses
include_cliffs with 422; get_report names the reform check for a run in any
status. The golden test that couldn't fail is replaced by one that fails if
the report run's result columns hold a cliff column again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.34711% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.63%. Comparing base (e87b0ea) to head (60fc6a1).
⚠️ Report is 2 commits behind head on v2plan/c1-37-cliff-results.

Files with missing lines Patch % Lines
...ices/v2/stored_reports/requester_views/services.py 94.28% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@                      Coverage Diff                       @@
##           v2plan/c1-37-cliff-results    #4329      +/-   ##
==============================================================
+ Coverage                       93.19%   93.63%   +0.43%     
==============================================================
  Files                             408      409       +1     
  Lines                           23297    23366      +69     
  Branches                         3648     3657       +9     
==============================================================
+ Hits                            21711    21878     +167     
+ Misses                            985      874     -111     
- Partials                          601      614      +13     
Flag Coverage Δ
integration 57.93% <95.04%> (+0.11%) ⬆️
unit 90.92% <71.90%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 5 commits November 18, 2026 18:26
The simulation service keeps refusing include_cliffs until it calculates
cliffs per simulation run, so the OpenAPI document must say so wherever it
tells callers to ask for them. Two client tests pin that service's own 422
bodies, which the Public API already reads and answers as 422. Test wording
stops saying the report run writer has already stopped storing cliffs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A simulation service that doesn't calculate cliffs per simulation run
refuses include_cliffs with 422 and creates or starts nothing, and that can
stay true for some time. The create and re-run routes, their 422 answers,
the re-run body, cliffs and the changelog fragment now say so, and that each
side stays not_calculated meanwhile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenAPI text must name include_cliffs: true, since false is accepted,
and say that a side with no calculation stays not_calculated, since a side
with a stored calculation keeps its status. Test support stops describing
sim's writers as if they had already changed: the fixture says it leaves out
the report run's cliff columns, which sim's writer can still fill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API text names include_cliffs: true, which is what such a service
refuses, and says only a side with no calculation stays not_calculated. A
re-run's request docstring no longer implies that every re-run without the
field runs: a stored report can still be refused for its own stored options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture and its callers say what they store rather than what sim's
writer stores, so the text holds before and after that writer changes, and
the create 422 keeps its sentence about the message beside the causes it
refers to. No behaviour changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 6 commits November 19, 2026 02:12
API v1 no longer uses report_runs' four cliff columns, but every whole-row
load of a report run still names them in its SQL. The release serving while
a later revision drops them would then fail each report read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ReportRun leaves baseline_cliff_gap, baseline_cliff_share, reform_cliff_gap
and reform_cliff_share out of its mapper, so API v1 neither selects nor
writes them. The table keeps the columns, their check constraint and their
grants, so there is no revision, and a release of this code keeps serving
report reads while a later revision drops them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unmapped columns declared as typed fields still look usable: a loaded report
run returns a Column object for them, and a value passed to the constructor
is dropped without an error. The table keeps them through a documented
direct SQLAlchemy category until a revision removes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The four columns move from typed SQLModel fields to __table_args__, so a
report run has no attribute or constructor field for them, and API v1's list
of direct SQLAlchemy uses names this one. The ReportRun description now says
this covers only these four: trigger is still mapped, so it needs an earlier
release of its own before a revision drops it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The earlier name contained the text the v2 model layer's rule forbids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The category no longer contains text the v2 model layer's rule forbids, and
it says API v1 stopped using these columns rather than that nothing does:
the simulation service still writes them until its own change deploys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in ddc422c, which changes #4327's wording from "the Public API"
to "API v1" and renames one privilege test. No file this branch changes
is touched, and revision bc334f9877b7 is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in e87b0ea, which merges master at 003ee49 (C1 · 36's
#4325: stored dynamics and revision 9d1d12a7ac58). Revision
bc334f9877b7 now revises 9d1d12a7ac58 and stays the only v2 head.
master's changes to files this branch also changes are route and
protocol wording and the dynamics protocols; none touches cliffs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant